Declare content signals, and advertise llms.txt in a Link header - #83
Merged
Conversation
… Link header Two gaps from the 2026-09-04 Cloudflare Radar agent-readiness scan, which put the property at Level 1 with Bot Access Control at 1/2 and Discoverability at 2/4. CONTENT SIGNALS. robots.txt now declares search=yes, ai-input=yes, ai-train=yes. All three are a decision rather than a default. This property exists to be found by counsel who increasingly ask an assistant before they ask a colleague, and each permission is a way these words reach that reader. Withholding training consent while asking to be cited would be taking one side of the same bargain. The declaration is repeated in all fourteen user-agent groups, and that repetition is load-bearing for the same reason the named crawler blocks are: a named group replaces the wildcard rather than adding to it. A Content-Signal written only under `User-agent: *` would be invisible to GPTBot, ClaudeBot, PerplexityBot and every other crawler named below it -- precisely the audience it addresses. Getting this half-right would have been worse than not doing it, because the file would have read as a declaration while making none. LINK HEADER. A new public/_headers advertises /llms.txt with RFC 8288 `rel="describedby"`, the registered relation for "the target describes this resource". An agent reading response headers finds the map without parsing HTML or guessing at a well-known path. Applied to /* rather than enumerated per route: an enumerated list silently misses the next route added, and the cost of the broader match is one short header on asset responses nobody reads. The Cloudflare adapter prepends its own /_astro/* cache rule to this file at build time, so both rules survive -- verified in dist/client/_headers. Not addressed, deliberately. Radar's Protocol Discovery is 0/9 -- API catalog, OAuth/OIDC, MCP server card, A2A agent card, agent skills, WebMCP, ARD. Every one of those describes an API or an agent that a site exposes. This is a brochure site for an expert-witness practice and exposes neither, so publishing any of them would be a fabrication. 0/9 is the correct score. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpEXBMByKmpV1f9G7UUZhF
The file was last touched 2026-08-08 and had drifted past usefulness: its status table still recorded that forensics.rootsystem.com does not resolve, which stopped being true when the route and DNS record landed on 2026-09-03. Everything else in it is either done or superseded. A stale roadmap is worse than no roadmap for the same reason a hand-kept dateModified is worse than none: it is a claim nobody verifies, and it is believed until someone acts on it. Planning now lives in Linear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CpEXBMByKmpV1f9G7UUZhF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The two actionable gaps from the Cloudflare Radar agent-readiness scan (
0f61525a), which put the property at Level 1 — Basic Web Presence, score 19: Bot Access Control 1/2, Discoverability 2/4.Content Signals
robots.txtnow declares:search=yesai-input=yesai-train=yesAll three yes, as a decision rather than a default. The property exists to be found by counsel who increasingly ask an assistant before they ask a colleague, and each permission is a way these words reach that reader. Withholding training consent while asking to be cited would be taking one side of the same bargain.
The repetition across all fourteen groups is load-bearing. Same rule as the named crawler blocks from #82: a named group replaces the wildcard rather than adding to it. A
Content-Signalwritten only underUser-agent: *would be invisible to GPTBot, ClaudeBot, PerplexityBot and every other crawler named beneath it — exactly the audience it addresses. Half-doing this would have been worse than skipping it: the file would read as a declaration while making none to anyone who matters.Link header
New
public/_headers:describedbyis the IANA-registered relation for "the target describes this resource," which is what/llms.txtis — the property's own map of itself. An agent reading response headers finds it without parsing HTML or guessing at well-known paths.Applied to
/*rather than enumerated per route: an enumerated list silently misses the next route someone adds, and the cost of the broader match is one short header on asset responses nobody reads.The Cloudflare adapter prepends its own
/_astro/*immutable-cache rule to this file at build time, so both rules survive — verified indist/client/_headersrather than assumed.Not addressed, deliberately
Radar's Protocol Discovery is 0/9: API catalog, OAuth/OIDC discovery, OAuth protected resource,
auth.md, MCP server card, A2A agent card, agent skills index, WebMCP, ARD manifest.Every one of those describes an API or an agent that a site exposes. This is a brochure site for an expert-witness practice and exposes neither. Publishing an MCP server card for a site with no MCP server is a fabrication, and the same reasoning that keeps
areaServedout of the schema applies. 0/9 is the correct score here, and chasing it would mean lying to a scanner.Markdown content negotiation — the one genuinely useful item in the remaining set — is filed separately for the roadmap rather than bundled here.
Verification
yarn buildclean; builtrobots.txtcarries 14User-agent:groups and 14Content-Signal:lines. Built_headerscontains both the adapter's rule and ours.The
Linkheader itself cannot be verified until this deploys —_headersis applied by Cloudflare at the edge, not by the build.🤖 Generated with Claude Code
https://claude.ai/code/session_01CpEXBMByKmpV1f9G7UUZhF